zfs(openzfs): cap the default ARC at half of RAM for app coexistence - #1487
zfs(openzfs): cap the default ARC at half of RAM for app coexistence#1487gburd wants to merge 2 commits into
Conversation
On OSv the ARC and the application share one physical address space. The existing arc_default_max() for a guest with >= 1 GiB of RAM returns MAX(allmem*5/8, allmem-1GiB), which on a large guest is nearly all of RAM (for example ~31 GiB of a 32 GiB guest). A memory-hungry application such as PostgreSQL, with several GiB of shared_buffers plus many concurrent backends, then has almost no headroom, and a burst (a large checkpoint stacking dirty ARC buffers on top of the backends working set) drives the guest out of memory and aborts it. Since OSv ZFS_MODULE_PARAM is a no-op (zfs_arc_max cannot be set at boot), cap the computed default at half of RAM for guests with more than 2 GiB, so the application always keeps at least half the guest memory. Small guests are unchanged (their fractions already leave room). A future zfs_arc_max wired to the boot cmdline could override this. Follow-on to the OpenZFS keystone (cloudius-systems#1423) and the runtime data pool (cloudius-systems#1478). Signed-off-by: Greg Burd <greg@burd.me>
…r OOM Companion to the half-RAM ARC cap: even with the ARC capped, at high concurrency (many PostgreSQL backends plus a large checkpoint and autovacuum worker forks) the OSv memory reclaimer can abort with out-of-memory while the ARC still holds evictable pages. OSv's reclaimer calls the ARC shrinker and, if that pass frees nothing, OOMs. The hard-mode shrinker only called arc_reduce_target_size_noshrink(), which lowers the target and wakes the async evict thread but returns before any page is freed, so under a burst the reclaimer saw zero freed bytes and aborted even though eviction was in flight. Add arc_reduce_target_size_shrink_sync(): it lowers the target the same way and then waits in arc_wait_for_eviction() until the requested amount is actually evicted (lax so it returns early once the ARC stops overflowing). The hard-mode shrinker now calls it, so the reclaimer gets real freed memory before deciding it cannot reclaim; soft mode stays asynchronous. The eviction waiter is already fork-COW coherent and the reclaimer thread may block. Measured on a large-working-set PostgreSQL-on-ZFS run: a virtual-user level that previously aborted with 'could not reclaim any further' now completes, riding through a large mid-run checkpoint, with no reclaimer OOM, no deadlock, and no throughput regression at the level below. NOTE: the arc.c hunk is applied via the vendored-edits patch; if a hunk offset drifts, apply with git apply --recount.
|
Added a second commit: a synchronous hard-mode ARC shrinker, the companion to the half-RAM cap. Even with the ARC capped, at high concurrency (many backends + a large checkpoint + autovacuum worker forks) OSv's reclaimer could abort out-of-memory while the ARC still held evictable pages, because the hard-mode shrinker only lowered the target and woke the async evict thread, returning before any page was freed. The new arc_reduce_target_size_shrink_sync() lowers the target and then waits in arc_wait_for_eviction() until the requested memory is actually evicted, so the reclaimer gets real freed memory before deciding it cannot reclaim; soft mode stays async. Measured: a virtual-user level that previously aborted with 'could not reclaim any further' now completes, riding through a large mid-run checkpoint, with no reclaimer OOM, no deadlock, and no throughput regression at the level below. Kept as part of this draft since both commits are ARC memory-pressure coexistence fixes for the same app-shares-RAM situation. |
Draft / follow-on to #1423 and #1478.
On OSv the ARC and the application share one physical address space.
arc_default_max()for a guest with >= 1 GiB of RAM returnsMAX(allmem*5/8, allmem-1GiB), which on a large guest is nearly all of RAM (about 31 GiB of a 32 GiB guest). A memory-hungry application such as PostgreSQL (multi-GiB shared_buffers plus many concurrent backends) then has almost no headroom, and a checkpoint burst stacking dirty ARC buffers on top of the backends' working set drives the guest OOM and aborts it. This showed up as an OOM at high concurrency on a 32 GiB guest.Since
ZFS_MODULE_PARAMis a no-op on OSv (zfs_arc_maxcannot be set at boot), this caps the computed default at half of RAM for guests with more than 2 GiB, so the application always keeps at least half the guest's memory. Small guests are unchanged.Draft because the proper long-term fix is to wire
zfs_arc_maxto the boot cmdline so this becomes an overridable default rather than a hard cap; kept as a draft PR so the tuning is not lost while the maintainer reviews the ZFS follow-on stack. Edits the trackedmodules/open_zfs/osv/module/os/osv/zfs/arc_os.cdirectly (the OSv platform layer is now source files on master, not a numbered patch).